{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "intro-title",
   "metadata": {},
   "source": [
    "# From Unstructured Text to Interactive Knowledge Graphs Using LLMs\n",
    "\n",
    "**Goal:** This notebook demonstrates a **highly granular, step-by-step process** to transform raw, unstructured text into a structured, interactive knowledge graph using Large Language Models (LLMs). We will extract factual information (SPO triples) and visualize the data transformations and final graph **directly within the notebook** at multiple stages.\n",
    "\n",
    "**Target Audience:** Beginners to Intermediate Python users interested in NLP, Knowledge Graphs, and LLMs, who want to see the data evolve at each step.\n",
    "\n",
    "**Approach:** We will break down the process into very small, logical steps. Each step will aim to perform a distinct action, followed by an output or visualization to show the immediate result. We'll use basic Python constructs and popular libraries, prioritizing clarity and step-by-step understanding over code conciseness."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "theory-kg",
   "metadata": {},
   "source": [
    "## Theory: What is a Knowledge Graph?\n",
    "\n",
    "A Knowledge Graph (KG) is a way to represent information as a network of entities and their relationships. Think of it like a structured database, but instead of tables, you have:\n",
    "\n",
    "*   **Nodes (or Entities):** These represent real-world objects, concepts, people, places, organizations, etc. (e.g., 'Marie Curie', 'Physics', 'Paris'). In our graph, each unique subject or object from our extracted facts will become a node.\n",
    "*   **Edges (or Relationships):** These represent the connections or interactions between entities. They typically have a direction and a label describing the relationship (e.g., 'Marie Curie' -- `won` --> 'Nobel Prize', 'Radium' -- `is element discovered by` --> 'Marie Curie'). In our graph, each predicate from our extracted facts defines an edge between the corresponding subject and object nodes.\n",
    "\n",
    "Knowledge graphs make it easier to understand complex connections, infer new information, and query data in intuitive ways. Visualizing the graph helps immensely in spotting patterns and understanding the overall structure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "theory-spo",
   "metadata": {},
   "source": [
    "## Theory: Subject-Predicate-Object (SPO) Triples\n",
    "\n",
    "The fundamental building block of many knowledge graphs derived from text is the **Subject-Predicate-Object (SPO)** triple. It's a simple structure that captures a single fact:\n",
    "\n",
    "*   **Subject:** The entity the statement is about (becomes a node).\n",
    "*   **Predicate:** The relationship or action connecting the subject and object (becomes the label on an edge).\n",
    "*   **Object:** The entity related to the subject via the predicate (becomes another node).\n",
    "\n",
    "**Example:** \"Marie Curie discovered Radium\" -> (`Marie Curie`, `discovered`, `Radium`).\n",
    "\n",
    "This translates to graph nodes and edges: `(Marie Curie) -[discovered]-> (Radium)`.\n",
    "\n",
    "LLMs help identify these triples by understanding language context."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup-intro",
   "metadata": {},
   "source": [
    "## Step 1: Setup - Installing Libraries\n",
    "\n",
    "First, we install the necessary Python libraries. We'll use:\n",
    "*   `openai`: For LLM API interaction.\n",
    "*   `networkx`: For graph data structures.\n",
    "*   `ipycytoscape`: For interactive in-notebook graph visualization.\n",
    "*   `ipywidgets`: Required by `ipycytoscape`.\n",
    "*   `pandas`: For displaying data nicely in tables.\n",
    "\n",
    "**Note:** You might need to restart the runtime/kernel after installation. Enable `ipywidgets` extension in classic Jupyter Notebook if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "setup-install",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install libraries (run this cell once)\n",
    "%pip install openai networkx \"ipycytoscape>=1.3.1\" ipywidgets pandas\n",
    "\n",
    "# If in classic Jupyter Notebook (not Lab), you might need to enable the widget extension:\n",
    "# jupyter nbextension enable --py widgetsnbextension\n",
    "\n",
    "# --- IMPORTANT: Restart the kernel/runtime after running this cell! ---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup-imports-title",
   "metadata": {},
   "source": [
    "## Step 2: Setup - Importing Libraries\n",
    "\n",
    "Now that the libraries are installed, we import the necessary components into our Python environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "setup-imports-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Libraries imported successfully.\n"
     ]
    }
   ],
   "source": [
    "import openai             # For LLM interaction\n",
    "import json               # For parsing LLM responses\n",
    "import networkx as nx     # For creating and managing the graph data structure\n",
    "import ipycytoscape       # For interactive in-notebook graph visualization\n",
    "import ipywidgets         # For interactive elements\n",
    "import pandas as pd       # For displaying data in tables\n",
    "import os                 # For accessing environment variables (safer for API keys)\n",
    "import math               # For basic math operations\n",
    "import re                 # For basic text cleaning (regular expressions)\n",
    "import warnings           # To suppress potential deprecation warnings\n",
    "\n",
    "# Configure settings for better display and fewer warnings\n",
    "warnings.filterwarnings('ignore', category=DeprecationWarning)\n",
    "pd.set_option('display.max_rows', 100) # Show more rows in pandas tables\n",
    "pd.set_option('display.max_colwidth', 150) # Show more text width in pandas tables\n",
    "\n",
    "print(\"Libraries imported successfully.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-config-intro-detail",
   "metadata": {},
   "source": [
    "## Step 3: Configure LLM Access\n",
    "\n",
    "We need to specify how to connect to the Large Language Model. This involves the API endpoint (URL) and the API key.\n",
    "\n",
    "**IMPORTANT SECURITY NOTE:** Use environment variables or a secure secrets manager for API keys. **Do not hardcode keys directly in the notebook.**\n",
    "\n",
    "**Environment Variable Setup (Example - run in your terminal *before* starting Jupyter):**\n",
    "```bash\n",
    "# For OpenAI\n",
    "export OPENAI_API_KEY='your_openai_api_key'\n",
    "\n",
    "# For Ollama (example)\n",
    "export OPENAI_API_KEY='ollama' # Or any non-empty string\n",
    "export OPENAI_API_BASE='http://localhost:11434/v1'\n",
    "\n",
    "# For Nebius AI (example)\n",
    "export OPENAI_API_KEY='your_nebius_api_key'\n",
    "export OPENAI_API_BASE='https://api.studio.nebius.com/v1/'\n",
    "```\n",
    "First, we'll define the model name we intend to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "llm-model-name",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intended LLM model: deepseek-ai/DeepSeek-V3\n"
     ]
    }
   ],
   "source": [
    "# --- Define LLM Model --- \n",
    "# Choose the model available at your configured endpoint.\n",
    "# Examples: 'gpt-4o', 'gpt-3.5-turbo', 'llama3', 'mistral', 'deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct', 'gemma'\n",
    "llm_model_name = \"deepseek-ai/DeepSeek-V3\" # <-- *** CHANGE THIS TO YOUR MODEL ***\n",
    "\n",
    "print(f\"Intended LLM model: {llm_model_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-config-keys",
   "metadata": {},
   "source": [
    "Now, let's retrieve the API key and base URL from environment variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "llm-config-retrieve-vars",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved API Key: Set\n",
      "Retrieved Base URL: https://api.studio.nebius.com/v1/\n"
     ]
    }
   ],
   "source": [
    "# --- Retrieve Credentials --- \n",
    "api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "base_url = os.getenv(\"OPENAI_API_BASE\") # Will be None if not set (e.g., for standard OpenAI)\n",
    "\n",
    "# --- FOR TESTING ONLY (Less Secure - Replace with Environment Variables) --- \n",
    "# Uncomment and set these lines ONLY if you cannot set environment variables easily.\n",
    "# api_key = \"YOUR_API_KEY_HERE\"  # <--- PASTE KEY HERE FOR TESTING ONLY\n",
    "# base_url = \"YOUR_API_BASE_URL_HERE\" # <--- PASTE BASE URL HERE (if needed)\n",
    "# Example for Nebius:\n",
    "# base_url=\"https://api.studio.nebius.com/v1/\"\n",
    "# api_key=\"YOUR_NEBIUS_KEY\"\n",
    "\n",
    "print(f\"Retrieved API Key: {'Set' if api_key else 'Not Set'}\")\n",
    "print(f\"Retrieved Base URL: {base_url if base_url else 'Not Set (will use default OpenAI)'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-config-client",
   "metadata": {},
   "source": [
    "Next, we validate the API key and initialize the `openai` client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "llm-config-init-client",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OpenAI client initialized successfully.\n"
     ]
    }
   ],
   "source": [
    "# --- Validate Key and Initialize Client --- \n",
    "if not api_key:\n",
    "    print(\"Error: OPENAI_API_KEY environment variable not set or key not provided directly.\")\n",
    "    print(\"Please set the environment variable (or uncomment/edit the test lines) and restart the kernel.\")\n",
    "    raise SystemExit(\"API Key configuration failed.\")\n",
    "else:\n",
    "    try:\n",
    "        client = openai.OpenAI(\n",
    "            base_url=base_url, # Pass None if not set, client handles default\n",
    "            api_key=api_key\n",
    "        )\n",
    "        print(\"OpenAI client initialized successfully.\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error initializing OpenAI client: {e}\")\n",
    "        print(\"Check your API key, base URL (if used), and network connection.\")\n",
    "        raise SystemExit(\"LLM client initialization failed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-config-params",
   "metadata": {},
   "source": [
    "Finally, let's define other LLM parameters like temperature and max tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "llm-config-set-params",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LLM Temperature set to: 0.0\n",
      "LLM Max Tokens set to: 4096\n"
     ]
    }
   ],
   "source": [
    "# --- Define LLM Call Parameters ---\n",
    "llm_temperature = 0.0 # Lower temperature for more deterministic, factual output. 0.0 is best for extraction.\n",
    "llm_max_tokens = 4096 # Max tokens for the LLM response (adjust based on model limits)\n",
    "\n",
    "print(f\"LLM Temperature set to: {llm_temperature}\")\n",
    "print(f\"LLM Max Tokens set to: {llm_max_tokens}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "input-text-intro-detail",
   "metadata": {},
   "source": [
    "## Step 4: Define Input Text\n",
    "\n",
    "Here, we define the raw, unstructured text we want to process. We'll use the Marie Curie biography."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "input-text-define",
   "metadata": {},
   "outputs": [],
   "source": [
    "unstructured_text = \"\"\"\n",
    "Marie Curie, born Maria Skłodowska in Warsaw, Poland, was a pioneering physicist and chemist.\n",
    "She conducted groundbreaking research on radioactivity. Together with her husband, Pierre Curie,\n",
    "she discovered the elements polonium and radium. Marie Curie was the first woman to win a Nobel Prize,\n",
    "the first person and only woman to win the Nobel Prize twice, and the only person to win the Nobel Prize\n",
    "in two different scientific fields. She won the Nobel Prize in Physics in 1903 with Pierre Curie\n",
    "and Henri Becquerel. Later, she won the Nobel Prize in Chemistry in 1911 for her work on radium and\n",
    "polonium. During World War I, she developed mobile radiography units, known as 'petites Curies',\n",
    "to provide X-ray services to field hospitals. Marie Curie died in 1934 from aplastic anemia, likely\n",
    "caused by her long-term exposure to radiation.\n",
    "\n",
    "Marie was born on November 7, 1867, to a family of teachers who valued education. She received her\n",
    "early schooling in Warsaw but moved to Paris in 1891 to continue her studies at the Sorbonne, where\n",
    "she earned degrees in physics and mathematics. She met Pierre Curie, a professor of physics, in 1894, \n",
    "and they married in 1895, beginning a productive scientific partnership. Following Pierre's tragic \n",
    "death in a street accident in 1906, Marie took over his teaching position, becoming the first female \n",
    "professor at the Sorbonne.\n",
    "\n",
    "The Curies' work on radioactivity was conducted in challenging conditions, in a poorly equipped shed \n",
    "with no proper ventilation, as they processed tons of pitchblende ore to isolate radium. Marie Curie\n",
    "established the Curie Institute in Paris, which became a major center for medical research. She had\n",
    "two daughters: Irène, who later won a Nobel Prize in Chemistry with her husband, and Eve, who became\n",
    "a writer. Marie's notebooks are still radioactive today and are kept in lead-lined boxes. Her legacy\n",
    "includes not only her scientific discoveries but also her role in breaking gender barriers in academia\n",
    "and science.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "input-text-display",
   "metadata": {},
   "source": [
    "Let's display the input text and some basic statistics about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "input-text-show",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Input Text Loaded ---\n",
      "\n",
      "Marie Curie, born Maria Skłodowska in Warsaw, Poland, was a pioneering physicist and chemist.\n",
      "She conducted groundbreaking research on radioactivity. Together with her husband, Pierre Curie,\n",
      "she discovered the elements polonium and radium. Marie Curie was the first woman to win a Nobel Prize,\n",
      "the first person and only woman to win the Nobel Prize twice, and the only person to win the Nobel Prize\n",
      "in two different scientific fields. She won the Nobel Prize in Physics in 1903 with Pierre Curie\n",
      "and Henri Becquerel. Later, she won the Nobel Prize in Chemistry in 1911 for her work on radium and\n",
      "polonium. During World War I, she developed mobile radiography units, known as 'petites Curies',\n",
      "to provide X-ray services to field hospitals. Marie Curie died in 1934 from aplastic anemia, likely\n",
      "caused by her long-term exposure to radiation.\n",
      "\n",
      "Marie was born on November 7, 1867, to a family of teachers who valued education. She received her\n",
      "early schooling in Warsaw but moved to Paris in 1891 to continue her studies at the Sorbonne, where\n",
      "she earned degrees in physics and mathematics. She met Pierre Curie, a professor of physics, in 1894, \n",
      "and they married in 1895, beginning a productive scientific partnership. Following Pierre's tragic \n",
      "death in a street accident in 1906, Marie took over his teaching position, becoming the first female \n",
      "professor at the Sorbonne.\n",
      "\n",
      "The Curies' work on radioactivity was conducted in challenging conditions, in a poorly equipped shed \n",
      "with no proper ventilation, as they processed tons of pitchblende ore to isolate radium. Marie Curie\n",
      "established the Curie Institute in Paris, which became a major center for medical research. She had\n",
      "two daughters: Irène, who later won a Nobel Prize in Chemistry with her husband, and Eve, who became\n",
      "a writer. Marie's notebooks are still radioactive today and are kept in lead-lined boxes. Her legacy\n",
      "includes not only her scientific discoveries but also her role in breaking gender barriers in academia\n",
      "and science.\n",
      "\n",
      "-------------------------\n",
      "Total characters: 1995\n",
      "Approximate word count: 324\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"--- Input Text Loaded ---\")\n",
    "print(unstructured_text)\n",
    "print(\"-\" * 25)\n",
    "# Basic stats visualization\n",
    "char_count = len(unstructured_text)\n",
    "word_count = len(unstructured_text.split())\n",
    "print(f\"Total characters: {char_count}\")\n",
    "print(f\"Approximate word count: {word_count}\")\n",
    "print(\"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chunking-intro-detail",
   "metadata": {},
   "source": [
    "## Step 5: Text Chunking (Optional but Recommended)\n",
    "\n",
    "LLMs have context limits. For longer texts, we need to break them into smaller chunks. We'll define the chunk size and overlap.\n",
    "\n",
    "*   **Chunk Size:** Max words per chunk.\n",
    "*   **Overlap:** Words shared between consecutive chunks to preserve context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "chunking-config",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chunk Size set to: 150 words\n",
      "Overlap set to: 30 words\n",
      "Chunking configuration is valid.\n"
     ]
    }
   ],
   "source": [
    "# --- Chunking Configuration ---\n",
    "chunk_size = 150  # Number of words per chunk (adjust as needed)\n",
    "overlap = 30     # Number of words to overlap (must be < chunk_size)\n",
    "\n",
    "print(f\"Chunk Size set to: {chunk_size} words\")\n",
    "print(f\"Overlap set to: {overlap} words\")\n",
    "\n",
    "# --- Basic Validation ---\n",
    "if overlap >= chunk_size and chunk_size > 0:\n",
    "    print(f\"Error: Overlap ({overlap}) must be smaller than chunk size ({chunk_size}).\")\n",
    "    raise SystemExit(\"Chunking configuration error.\")\n",
    "else:\n",
    "    print(\"Chunking configuration is valid.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chunking-split-words",
   "metadata": {},
   "source": [
    "First, let's split the input text into a list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "chunking-get-words",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text split into 324 words.\n",
      "First 20 words: ['Marie', 'Curie,', 'born', 'Maria', 'Skłodowska', 'in', 'Warsaw,', 'Poland,', 'was', 'a', 'pioneering', 'physicist', 'and', 'chemist.', 'She', 'conducted', 'groundbreaking', 'research', 'on', 'radioactivity.']\n"
     ]
    }
   ],
   "source": [
    "words = unstructured_text.split()\n",
    "total_words = len(words)\n",
    "\n",
    "print(f\"Text split into {total_words} words.\")\n",
    "# Visualize the first 20 words\n",
    "print(f\"First 20 words: {words[:20]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chunking-perform",
   "metadata": {},
   "source": [
    "Now, we'll perform the chunking based on the configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "chunking-do-chunk",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting chunking process...\n",
      "\n",
      "Text successfully split into 3 chunks.\n"
     ]
    }
   ],
   "source": [
    "chunks = []\n",
    "start_index = 0\n",
    "chunk_number = 1\n",
    "\n",
    "print(f\"Starting chunking process...\")\n",
    "\n",
    "while start_index < total_words:\n",
    "    end_index = min(start_index + chunk_size, total_words)\n",
    "    chunk_text = \" \".join(words[start_index:end_index])\n",
    "    chunks.append({\"text\": chunk_text, \"chunk_number\": chunk_number})\n",
    "    \n",
    "    # print(f\"  Created chunk {chunk_number}: words {start_index} to {end_index-1}\") # Uncomment for detailed log\n",
    "    \n",
    "    # Calculate the start of the next chunk\n",
    "    next_start_index = start_index + chunk_size - overlap\n",
    "    \n",
    "    # Ensure progress is made\n",
    "    if next_start_index <= start_index:\n",
    "        if end_index == total_words:\n",
    "             break # Already processed the last part\n",
    "        next_start_index = start_index + 1 \n",
    "         \n",
    "    start_index = next_start_index\n",
    "    chunk_number += 1\n",
    "    \n",
    "    # Safety break (optional)\n",
    "    if chunk_number > total_words: # Simple safety\n",
    "        print(\"Warning: Chunking loop exceeded total word count, breaking.\")\n",
    "        break\n",
    "\n",
    "print(f\"\\nText successfully split into {len(chunks)} chunks.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chunking-visualize",
   "metadata": {},
   "source": [
    "Let's visualize the created chunks using Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "chunking-display",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Chunk Details ---\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>chunk_number</th>\n",
       "      <th>word_count</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>150</td>\n",
       "      <td>Marie Curie, born Maria Skłodowska in Warsaw, Poland, was a pioneering physicist and chemist. She conducted groundbreaking research on radioactivi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>field hospitals. Marie Curie died in 1934 from aplastic anemia, likely caused by her long-term exposure to radiation. Marie was born on November 7...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>84</td>\n",
       "      <td>with no proper ventilation, as they processed tons of pitchblende ore to isolate radium. Marie Curie established the Curie Institute in Paris, whi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   chunk_number  word_count  \\\n",
       "0             1         150   \n",
       "1             2         150   \n",
       "2             3          84   \n",
       "\n",
       "                                                                                                                                                    text  \n",
       "0  Marie Curie, born Maria Skłodowska in Warsaw, Poland, was a pioneering physicist and chemist. She conducted groundbreaking research on radioactivi...  \n",
       "1  field hospitals. Marie Curie died in 1934 from aplastic anemia, likely caused by her long-term exposure to radiation. Marie was born on November 7...  \n",
       "2  with no proper ventilation, as they processed tons of pitchblende ore to isolate radium. Marie Curie established the Curie Institute in Paris, whi...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"--- Chunk Details ---\")\n",
    "if chunks:\n",
    "    # Create a DataFrame for better visualization\n",
    "    chunks_df = pd.DataFrame(chunks)\n",
    "    chunks_df['word_count'] = chunks_df['text'].apply(lambda x: len(x.split()))\n",
    "    display(chunks_df[['chunk_number', 'word_count', 'text']])\n",
    "else:\n",
    "    print(\"No chunks were created (text might be shorter than chunk size).\")\n",
    "print(\"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prompt-intro-detail",
   "metadata": {},
   "source": [
    "## Step 6: Define the LLM Prompt for Extraction\n",
    "\n",
    "This is a critical step. We need to carefully instruct the LLM to extract SPO triples in a specific JSON format. We'll define a system prompt (role) and a user prompt template (instructions).\n",
    "\n",
    "**Key Instructions Emphasized:**\n",
    "*   Extract `Subject-Predicate-Object` triples.\n",
    "*   Output *only* a valid JSON array of objects.\n",
    "*   Each object must have `\"subject\"`, `\"predicate\"`, `\"object\"` keys.\n",
    "*   Predicates should be concise (1-3 words).\n",
    "*   All output values must be lowercase.\n",
    "*   Resolve pronouns to specific entity names.\n",
    "*   No extra text, explanations, or markdown code fences around the JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "prompt-define",
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- System Prompt: Sets the context/role for the LLM --- \n",
    "extraction_system_prompt = \"\"\"\n",
    "You are an AI expert specialized in knowledge graph extraction. \n",
    "Your task is to identify and extract factual Subject-Predicate-Object (SPO) triples from the given text.\n",
    "Focus on accuracy and adhere strictly to the JSON output format requested in the user prompt.\n",
    "Extract core entities and the most direct relationship.\n",
    "\"\"\"\n",
    "\n",
    "# --- User Prompt Template: Contains specific instructions and the text --- \n",
    "extraction_user_prompt_template = \"\"\"\n",
    "Please extract Subject-Predicate-Object (S-P-O) triples from the text below.\n",
    "\n",
    "**VERY IMPORTANT RULES:**\n",
    "1.  **Output Format:** Respond ONLY with a single, valid JSON array. Each element MUST be an object with keys \"subject\", \"predicate\", \"object\".\n",
    "2.  **JSON Only:** Do NOT include any text before or after the JSON array (e.g., no 'Here is the JSON:' or explanations). Do NOT use markdown ```json ... ``` tags.\n",
    "3.  **Concise Predicates:** Keep the 'predicate' value concise (1-3 words, ideally 1-2). Use verbs or short verb phrases (e.g., 'discovered', 'was born in', 'won').\n",
    "4.  **Lowercase:** ALL values for 'subject', 'predicate', and 'object' MUST be lowercase.\n",
    "5.  **Pronoun Resolution:** Replace pronouns (she, he, it, her, etc.) with the specific lowercase entity name they refer to based on the text context (e.g., 'marie curie').\n",
    "6.  **Specificity:** Capture specific details (e.g., 'nobel prize in physics' instead of just 'nobel prize' if specified).\n",
    "7.  **Completeness:** Extract all distinct factual relationships mentioned.\n",
    "\n",
    "**Text to Process:**\n",
    "```text\n",
    "{text_chunk}\n",
    "```\n",
    "\n",
    "**Required JSON Output Format Example:**\n",
    "[\n",
    "  {{ \"subject\": \"marie curie\", \"predicate\": \"discovered\", \"object\": \"radium\" }},\n",
    "  {{ \"subject\": \"marie curie\", \"predicate\": \"won\", \"object\": \"nobel prize in physics\" }}\n",
    "]\n",
    "\n",
    "**Your JSON Output (MUST start with '[' and end with ']'):**\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prompt-display",
   "metadata": {},
   "source": [
    "Let's display the prompts we've defined to verify them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "prompt-show",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- System Prompt ---\n",
      "\n",
      "You are an AI expert specialized in knowledge graph extraction. \n",
      "Your task is to identify and extract factual Subject-Predicate-Object (SPO) triples from the given text.\n",
      "Focus on accuracy and adhere strictly to the JSON output format requested in the user prompt.\n",
      "Extract core entities and the most direct relationship.\n",
      "\n",
      "\n",
      "-------------------------\n",
      "\n",
      "--- User Prompt Template (Structure) ---\n",
      "\n",
      "Please extract Subject-Predicate-Object (S-P-O) triples from the text below.\n",
      "\n",
      "**VERY IMPORTANT RULES:**\n",
      "1.  **Output Format:** Respond ONLY with a single, valid JSON array. Each element MUST be an object with keys \"subject\", \"predicate\", \"object\".\n",
      "2.  **JSON Only:** Do NOT include any text before or after the JSON array (e.g., no 'Here is the JSON:' or explanations). Do NOT use markdown ```json ... ``` tags.\n",
      "3.  **Concise Predicates:** Keep the 'predicate' value concise (1-3 words, ideally 1-2). Use verbs or short verb phrases (e.g., 'discovered', 'was born in', 'won').\n",
      "4.  **Lowercase:** ALL values for 'subject', 'predicate', and 'object' MUST be lowercase.\n",
      "5.  **Pronoun Resolution:** Replace pronouns (she, he, it, her, etc.) with the specific lowercase entity name they refer to based on the text context (e.g., 'marie curie').\n",
      "6.  **Specificity:** Capture specific details (e.g., 'nobel prize in physics' instead of just 'nobel prize' if specified).\n",
      "7.  **Completeness:** Extract all distinct factual relationships mentioned.\n",
      "\n",
      "**Text to Process:**\n",
      "```text\n",
      "[... text chunk goes here ...]\n",
      "```\n",
      "\n",
      "**Required JSON Output Format Example:**\n",
      "[\n",
      "  {{ \"subject\": \"marie curie\", \"predicate\": \"discovered\", \"object\": \"radium\" }},\n",
      "  {{ \"subject\": \"marie curie\", \"predicate\": \"won\", \"object\": \"nobel prize in physics\" }}\n",
      "]\n",
      "\n",
      "**Your JSON Output (MUST start with '[' and end with ']'):**\n",
      "\n",
      "\n",
      "-------------------------\n",
      "\n",
      "--- Example Filled User Prompt (for Chunk 1) ---\n",
      "\n",
      "Please extract Subject-Predicate-Object (S-P-O) triples from the text below.\n",
      "\n",
      "**VERY IMPORTANT RULES:**\n",
      "1.  **Output Format:** Respond ONLY with a single, valid JSON array. Each element MUST be an object with keys \"subject\", \"predicate\", \"object\".\n",
      "2.  **JSON Only:** Do NOT include any text before or after the JSON array (e.g., no 'Here is the JSON:' or explanations). Do NOT use markdown ```json ... ``` tags.\n",
      "3.  **Concise Predicates:** Keep the 'predicate' value concise (1-3 words, ideally 1-2). Use verbs or short verb phrases (e.g., 'discovered', 'was born in', 'won').\n",
      "4.  **Lowercase:** ALL\n",
      "[... rest of chunk text ...]\n",
      "\"predicate\": \"discovered\", \"object\": \"radium\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"won\", \"object\": \"nobel prize in physics\" }\n",
      "]\n",
      "\n",
      "**Your JSON Output (MUST start with '[' and end with ']'):**\n",
      "\n",
      "\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "print(\"--- System Prompt ---\")\n",
    "print(extraction_system_prompt)\n",
    "print(\"\\n\" + \"-\" * 25 + \"\\n\")\n",
    "\n",
    "print(\"--- User Prompt Template (Structure) ---\")\n",
    "# Show structure, replacing the placeholder for clarity\n",
    "print(extraction_user_prompt_template.replace(\"{text_chunk}\", \"[... text chunk goes here ...]\"))\n",
    "print(\"\\n\" + \"-\" * 25 + \"\\n\")\n",
    "\n",
    "# Show an example of the *actual* prompt that will be sent for the first chunk\n",
    "print(\"--- Example Filled User Prompt (for Chunk 1) ---\")\n",
    "if chunks:\n",
    "    example_filled_prompt = extraction_user_prompt_template.format(text_chunk=chunks[0]['text'])\n",
    "    # Displaying a limited portion for brevity\n",
    "    print(example_filled_prompt[:600] + \"\\n[... rest of chunk text ...]\\n\" + example_filled_prompt[-200:])\n",
    "else:\n",
    "    print(\"No chunks available to create an example filled prompt.\")\n",
    "print(\"\\n\" + \"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-extract-intro-detail",
   "metadata": {},
   "source": [
    "## Step 7: LLM Interaction - Extracting Triples (Chunk by Chunk)\n",
    "\n",
    "Now we loop through each text chunk, send it to the LLM with our prompts, and attempt to parse the expected JSON output. We will show the process for each chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "llm-extract-init",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting triple extraction from 3 chunks using model 'deepseek-ai/DeepSeek-V3'...\n"
     ]
    }
   ],
   "source": [
    "# Initialize lists to store results and failures\n",
    "all_extracted_triples = []\n",
    "failed_chunks = []\n",
    "\n",
    "print(f\"Starting triple extraction from {len(chunks)} chunks using model '{llm_model_name}'...\")\n",
    "# We will process chunks one by one in the following cells."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-extract-loop-start",
   "metadata": {},
   "source": [
    "### Processing Chunk 1 (Example - loop structure will handle all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "llm-extract-loop-body",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Processing Chunk 1/3 --- \n",
      "1. Formatting User Prompt...\n",
      "2. Sending request to LLM...\n",
      "   LLM response received.\n",
      "3. Extracting raw response content...\n",
      "--- Raw LLM Output (Chunk {chunk_num}) ---\n",
      "[\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"born as\", \"object\": \"maria skłodowska\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"born in\", \"object\": \"warsaw, poland\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"was\", \"object\": \"physicist\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"was\", \"object\": \"chemist\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"conducted\", \"object\": \"research on radioactivity\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"discovered\", \"object\": \"polonium\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"discovered\", \"object\": \"radium\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"was\", \"object\": \"first woman to win nobel prize\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"was\", \"object\": \"first person to win nobel prize twice\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"was\", \"object\": \"only woman to win nobel prize twice\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"was\", \"object\": \"only person to win nobel prize in two scientific fields\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"won\", \"object\": \"nobel prize in physics\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"won\", \"object\": \"nobel prize in chemistry\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"developed\", \"object\": \"mobile radiography units\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"died in\", \"object\": \"1934\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"died from\", \"object\": \"aplastic anemia\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"born on\", \"object\": \"november 7, 1867\" },\n",
      "  { \"subject\": \"marie curie\", \"predicate\": \"born to\", \"object\": \"family of teachers\" }\n",
      "]\n",
      "--------------------\n",
      "4. Attempting to parse JSON from response...\n",
      "   Successfully parsed JSON list directly.\n",
      "--- Parsed JSON Data (Chunk {chunk_num}) ---\n",
      "[\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"born as\",\n",
      "    \"object\": \"maria sk\\u0142odowska\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"born in\",\n",
      "    \"object\": \"warsaw, poland\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"was\",\n",
      "    \"object\": \"physicist\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"was\",\n",
      "    \"object\": \"chemist\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"conducted\",\n",
      "    \"object\": \"research on radioactivity\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"discovered\",\n",
      "    \"object\": \"polonium\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"discovered\",\n",
      "    \"object\": \"radium\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"was\",\n",
      "    \"object\": \"first woman to win nobel prize\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"was\",\n",
      "    \"object\": \"first person to win nobel prize twice\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"was\",\n",
      "    \"object\": \"only woman to win nobel prize twice\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"was\",\n",
      "    \"object\": \"only person to win nobel prize in two scientific fields\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"won\",\n",
      "    \"object\": \"nobel prize in physics\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"won\",\n",
      "    \"object\": \"nobel prize in chemistry\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"developed\",\n",
      "    \"object\": \"mobile radiography units\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"died in\",\n",
      "    \"object\": \"1934\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"died from\",\n",
      "    \"object\": \"aplastic anemia\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"born on\",\n",
      "    \"object\": \"november 7, 1867\"\n",
      "  },\n",
      "  {\n",
      "    \"subject\": \"marie curie\",\n",
      "    \"predicate\": \"born to\",\n",
      "    \"object\": \"family of teachers\"\n",
      "  }\n",
      "]\n",
      "--------------------\n",
      "5. Validating structure and extracting triples...\n",
      "   Found 18 valid triples in this chunk.\n",
      "--- Valid Triples Extracted (Chunk 1) ---\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject</th>\n",
       "      <th>predicate</th>\n",
       "      <th>object</th>\n",
       "      <th>chunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born as</td>\n",
       "      <td>maria skłodowska</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born in</td>\n",
       "      <td>warsaw, poland</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>physicist</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>chemist</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>conducted</td>\n",
       "      <td>research on radioactivity</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>discovered</td>\n",
       "      <td>polonium</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>discovered</td>\n",
       "      <td>radium</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>first woman to win nobel prize</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>first person to win nobel prize twice</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>only woman to win nobel prize twice</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>only person to win nobel prize in two scientific fields</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>won</td>\n",
       "      <td>nobel prize in physics</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>won</td>\n",
       "      <td>nobel prize in chemistry</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>developed</td>\n",
       "      <td>mobile radiography units</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>died in</td>\n",
       "      <td>1934</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>died from</td>\n",
       "      <td>aplastic anemia</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born on</td>\n",
       "      <td>november 7, 1867</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born to</td>\n",
       "      <td>family of teachers</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        subject   predicate  \\\n",
       "0   marie curie     born as   \n",
       "1   marie curie     born in   \n",
       "2   marie curie         was   \n",
       "3   marie curie         was   \n",
       "4   marie curie   conducted   \n",
       "5   marie curie  discovered   \n",
       "6   marie curie  discovered   \n",
       "7   marie curie         was   \n",
       "8   marie curie         was   \n",
       "9   marie curie         was   \n",
       "10  marie curie         was   \n",
       "11  marie curie         won   \n",
       "12  marie curie         won   \n",
       "13  marie curie   developed   \n",
       "14  marie curie     died in   \n",
       "15  marie curie   died from   \n",
       "16  marie curie     born on   \n",
       "17  marie curie     born to   \n",
       "\n",
       "                                                     object  chunk  \n",
       "0                                          maria skłodowska      1  \n",
       "1                                            warsaw, poland      1  \n",
       "2                                                 physicist      1  \n",
       "3                                                   chemist      1  \n",
       "4                                 research on radioactivity      1  \n",
       "5                                                  polonium      1  \n",
       "6                                                    radium      1  \n",
       "7                            first woman to win nobel prize      1  \n",
       "8                     first person to win nobel prize twice      1  \n",
       "9                       only woman to win nobel prize twice      1  \n",
       "10  only person to win nobel prize in two scientific fields      1  \n",
       "11                                   nobel prize in physics      1  \n",
       "12                                 nobel prize in chemistry      1  \n",
       "13                                 mobile radiography units      1  \n",
       "14                                                     1934      1  \n",
       "15                                          aplastic anemia      1  \n",
       "16                                         november 7, 1867      1  \n",
       "17                                       family of teachers      1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------\n",
      "--- Running Total Triples Extracted: 18 --- \n",
      "--- Failed Chunks So Far: 0 --- \n",
      "\n",
      "Finished processing this chunk.\n"
     ]
    }
   ],
   "source": [
    "# --- This cell represents the core logic inside the loop for ONE chunk --- \n",
    "# --- In a real run, this logic would be in a loop like the original notebook --- \n",
    "# --- We show it step-by-step for the first chunk for clarity --- \n",
    "\n",
    "chunk_index = 0 # For demonstration, we process only the first chunk here\n",
    "\n",
    "if chunk_index < len(chunks):\n",
    "    chunk_info = chunks[chunk_index]\n",
    "    chunk_text = chunk_info['text']\n",
    "    chunk_num = chunk_info['chunk_number']\n",
    "    \n",
    "    print(f\"\\n--- Processing Chunk {chunk_num}/{len(chunks)} --- \")\n",
    "    \n",
    "    # 1. Format the User Prompt\n",
    "    print(\"1. Formatting User Prompt...\")\n",
    "    user_prompt = extraction_user_prompt_template.format(text_chunk=chunk_text)\n",
    "    # print(f\"   Formatted Prompt (Snippet): {user_prompt[:200]}...{user_prompt[-100:]}\") # Optional: View prompt\n",
    "    \n",
    "    llm_output = None\n",
    "    error_message = None\n",
    "    \n",
    "    try:\n",
    "        # 2. Make the API Call\n",
    "        print(\"2. Sending request to LLM...\")\n",
    "        response = client.chat.completions.create(\n",
    "            model=llm_model_name,\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": extraction_system_prompt},\n",
    "                {\"role\": \"user\", \"content\": user_prompt}\n",
    "            ],\n",
    "            temperature=llm_temperature,\n",
    "            max_tokens=llm_max_tokens,\n",
    "            # Request JSON output format - helps models that support it\n",
    "            response_format={ \"type\": \"json_object\" }, \n",
    "        )\n",
    "        print(\"   LLM response received.\")\n",
    "        \n",
    "        # 3. Extract Raw Response Content\n",
    "        print(\"3. Extracting raw response content...\")\n",
    "        llm_output = response.choices[0].message.content.strip()\n",
    "        print(\"--- Raw LLM Output (Chunk {chunk_num}) ---\")\n",
    "        print(llm_output)\n",
    "        print(\"-\" * 20)\n",
    "\n",
    "    except Exception as e:\n",
    "        error_message = str(e)\n",
    "        print(f\"   ERROR during API call: {error_message}\")\n",
    "        failed_chunks.append({'chunk_number': chunk_num, 'error': f'API/Processing Error: {error_message}', 'response': ''})\n",
    "\n",
    "    # 4. Parse JSON (if API call succeeded)\n",
    "    parsed_json = None\n",
    "    parsing_error = None\n",
    "    if llm_output is not None:\n",
    "        print(\"4. Attempting to parse JSON from response...\")\n",
    "        try:\n",
    "            # Strategy 1: Direct parsing (ideal)\n",
    "            parsed_data = json.loads(llm_output)\n",
    "\n",
    "            # Handle if response_format={'type':'json_object'} returns a dict containing the list\n",
    "            if isinstance(parsed_data, dict):\n",
    "                print(\"   Detected dictionary response, attempting to extract list...\")\n",
    "                list_values = [v for v in parsed_data.values() if isinstance(v, list)]\n",
    "                if len(list_values) == 1:\n",
    "                    parsed_json = list_values[0]\n",
    "                    print(\"      Successfully extracted list from dictionary.\")\n",
    "                else:\n",
    "                    raise ValueError(\"JSON object received, but doesn't contain a single list of triples.\")\n",
    "            elif isinstance(parsed_data, list):\n",
    "                parsed_json = parsed_data\n",
    "                print(\"   Successfully parsed JSON list directly.\")\n",
    "            else:\n",
    "                 raise ValueError(\"Parsed JSON is not a list or expected dictionary wrapper.\")\n",
    "\n",
    "        except json.JSONDecodeError as json_err:\n",
    "            parsing_error = f\"JSONDecodeError: {json_err}. Trying regex fallback...\"\n",
    "            print(f\"   {parsing_error}\")\n",
    "            # Strategy 2: Regex fallback for arrays potentially wrapped in text/markdown\n",
    "            match = re.search(r'^\\s*(\\[.*?\\])\\s*$', llm_output, re.DOTALL)\n",
    "            if match:\n",
    "                json_string_extracted = match.group(1)\n",
    "                print(\"      Regex found potential JSON array structure.\")\n",
    "                try:\n",
    "                    parsed_json = json.loads(json_string_extracted)\n",
    "                    print(\"      Successfully parsed JSON from regex extraction.\")\n",
    "                    parsing_error = None # Clear previous error\n",
    "                except json.JSONDecodeError as nested_err:\n",
    "                    parsing_error = f\"JSONDecodeError after regex: {nested_err}\"\n",
    "                    print(f\"      ERROR: Regex content is not valid JSON: {nested_err}\")\n",
    "            else:\n",
    "                 parsing_error = \"JSONDecodeError and Regex fallback failed.\"\n",
    "                 print(\"      ERROR: Regex could not find JSON array structure.\")\n",
    "                 \n",
    "        except ValueError as val_err:\n",
    "             parsing_error = f\"ValueError: {val_err}\" # Catches issues with unexpected structure\n",
    "             print(f\"   ERROR: {parsing_error}\")\n",
    "\n",
    "        # --- Show Parsed Result (or error) ---\n",
    "        if parsed_json is not None:\n",
    "            print(\"--- Parsed JSON Data (Chunk {chunk_num}) ---\")\n",
    "            print(json.dumps(parsed_json, indent=2)) # Pretty print the JSON\n",
    "            print(\"-\" * 20)\n",
    "        else:\n",
    "            print(f\"--- JSON Parsing FAILED (Chunk {chunk_num}) --- \")\n",
    "            print(f\"   Final Parsing Error: {parsing_error}\")\n",
    "            print(\"-\" * 20)\n",
    "            failed_chunks.append({'chunk_number': chunk_num, 'error': f'Parsing Failed: {parsing_error}', 'response': llm_output})\n",
    "\n",
    "    # 5. Validate and Store Triples (if parsing succeeded)\n",
    "    if parsed_json is not None:\n",
    "        print(\"5. Validating structure and extracting triples...\")\n",
    "        valid_triples_in_chunk = []\n",
    "        invalid_entries = []\n",
    "        if isinstance(parsed_json, list):\n",
    "            for item in parsed_json:\n",
    "                if isinstance(item, dict) and all(k in item for k in ['subject', 'predicate', 'object']):\n",
    "                    # Basic check: ensure values are strings (can be refined)\n",
    "                    if all(isinstance(item[k], str) for k in ['subject', 'predicate', 'object']):\n",
    "                        item['chunk'] = chunk_num # Add source chunk info\n",
    "                        valid_triples_in_chunk.append(item)\n",
    "                    else:\n",
    "                        invalid_entries.append({'item': item, 'reason': 'Non-string value'}) \n",
    "                else:\n",
    "                    invalid_entries.append({'item': item, 'reason': 'Incorrect structure/keys'})\n",
    "        else:\n",
    "            print(\"   ERROR: Parsed data is not a list, cannot extract triples.\")\n",
    "            invalid_entries.append({'item': parsed_json, 'reason': 'Not a list'})\n",
    "            # Also add to failed chunks if the overall structure was wrong\n",
    "            if not any(fc['chunk_number'] == chunk_num for fc in failed_chunks):\n",
    "                 failed_chunks.append({'chunk_number': chunk_num, 'error': 'Parsed data not a list', 'response': llm_output})\n",
    "        \n",
    "        # --- Show Validation Results --- \n",
    "        print(f\"   Found {len(valid_triples_in_chunk)} valid triples in this chunk.\")\n",
    "        if invalid_entries:\n",
    "             print(f\"   Skipped {len(invalid_entries)} invalid entries.\")\n",
    "             # print(f\"   Invalid entries details: {invalid_entries}\") # Uncomment for debugging\n",
    "             \n",
    "        # --- Display Valid Triples from this Chunk --- \n",
    "        if valid_triples_in_chunk:\n",
    "             print(f\"--- Valid Triples Extracted (Chunk {chunk_num}) ---\")\n",
    "             display(pd.DataFrame(valid_triples_in_chunk))\n",
    "             print(\"-\" * 20)\n",
    "             # Add to the main list\n",
    "             all_extracted_triples.extend(valid_triples_in_chunk)\n",
    "        else:\n",
    "             print(f\"--- No valid triples extracted from this chunk. ---\")\n",
    "             print(\"-\" * 20)\n",
    "\n",
    "    # --- Update Running Total (Visual Feedback) ---\n",
    "    print(f\"--- Running Total Triples Extracted: {len(all_extracted_triples)} --- \")\n",
    "    print(f\"--- Failed Chunks So Far: {len(failed_chunks)} --- \")\n",
    "        \n",
    "else:\n",
    "    print(f\"Chunk index {chunk_index} is out of bounds (Total chunks: {len(chunks)}). Skipping.\")\n",
    "\n",
    "print(\"\\nFinished processing this chunk.\")\n",
    "# --- IMPORTANT: In a full run, you would uncomment the loop in the original notebook --- \n",
    "# --- and remove the `chunk_index = 0` line to process ALL chunks. --- "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "llm-extract-summary-title",
   "metadata": {},
   "source": [
    "### Extraction Summary (After Processing All Chunks)\n",
    "\n",
    "**(Note:** The previous cell only processed *one* chunk for demonstration. In a full run, the loop would process all chunks. The summary below reflects the state *after* the demonstrated single chunk processing. Run the full loop from the original notebook to get the complete results).**\n",
    "\n",
    "Let's summarize the extraction results and display all accumulated triples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "llm-extract-summary-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Overall Extraction Summary ---\n",
      "Total chunks defined: 3\n",
      "Chunks processed (attempted): 3\n",
      "Total valid triples extracted across all processed chunks: 18\n",
      "Number of chunks that failed API call or parsing: 0\n",
      "-------------------------\n",
      "\n",
      "--- All Extracted Triples (Before Normalization) ---\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject</th>\n",
       "      <th>predicate</th>\n",
       "      <th>object</th>\n",
       "      <th>chunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born as</td>\n",
       "      <td>maria skłodowska</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born in</td>\n",
       "      <td>warsaw, poland</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>physicist</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>chemist</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>conducted</td>\n",
       "      <td>research on radioactivity</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>discovered</td>\n",
       "      <td>polonium</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>discovered</td>\n",
       "      <td>radium</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>first woman to win nobel prize</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>first person to win nobel prize twice</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>only woman to win nobel prize twice</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>only person to win nobel prize in two scientific fields</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>won</td>\n",
       "      <td>nobel prize in physics</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>won</td>\n",
       "      <td>nobel prize in chemistry</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>developed</td>\n",
       "      <td>mobile radiography units</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>died in</td>\n",
       "      <td>1934</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>died from</td>\n",
       "      <td>aplastic anemia</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born on</td>\n",
       "      <td>november 7, 1867</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born to</td>\n",
       "      <td>family of teachers</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        subject   predicate  \\\n",
       "0   marie curie     born as   \n",
       "1   marie curie     born in   \n",
       "2   marie curie         was   \n",
       "3   marie curie         was   \n",
       "4   marie curie   conducted   \n",
       "5   marie curie  discovered   \n",
       "6   marie curie  discovered   \n",
       "7   marie curie         was   \n",
       "8   marie curie         was   \n",
       "9   marie curie         was   \n",
       "10  marie curie         was   \n",
       "11  marie curie         won   \n",
       "12  marie curie         won   \n",
       "13  marie curie   developed   \n",
       "14  marie curie     died in   \n",
       "15  marie curie   died from   \n",
       "16  marie curie     born on   \n",
       "17  marie curie     born to   \n",
       "\n",
       "                                                     object  chunk  \n",
       "0                                          maria skłodowska      1  \n",
       "1                                            warsaw, poland      1  \n",
       "2                                                 physicist      1  \n",
       "3                                                   chemist      1  \n",
       "4                                 research on radioactivity      1  \n",
       "5                                                  polonium      1  \n",
       "6                                                    radium      1  \n",
       "7                            first woman to win nobel prize      1  \n",
       "8                     first person to win nobel prize twice      1  \n",
       "9                       only woman to win nobel prize twice      1  \n",
       "10  only person to win nobel prize in two scientific fields      1  \n",
       "11                                   nobel prize in physics      1  \n",
       "12                                 nobel prize in chemistry      1  \n",
       "13                                 mobile radiography units      1  \n",
       "14                                                     1934      1  \n",
       "15                                          aplastic anemia      1  \n",
       "16                                         november 7, 1867      1  \n",
       "17                                       family of teachers      1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "# --- Summary of Extraction (Reflecting state after the single chunk demo) ---\n",
    "print(f\"\\n--- Overall Extraction Summary ---\")\n",
    "print(f\"Total chunks defined: {len(chunks)}\")\n",
    "processed_chunks = len(chunks) - len(failed_chunks) # Approximation if loop isn't run fully\n",
    "print(f\"Chunks processed (attempted): {processed_chunks + len(failed_chunks)}\") # Chunks we looped through\n",
    "print(f\"Total valid triples extracted across all processed chunks: {len(all_extracted_triples)}\")\n",
    "print(f\"Number of chunks that failed API call or parsing: {len(failed_chunks)}\")\n",
    "\n",
    "if failed_chunks:\n",
    "    print(\"\\nDetails of Failed Chunks:\")\n",
    "    for failure in failed_chunks:\n",
    "        print(f\"  Chunk {failure['chunk_number']}: Error: {failure['error']}\")\n",
    "        # print(f\"    Response (start): {failure.get('response', '')[:100]}...\") # Uncomment for more detail\n",
    "print(\"-\" * 25)\n",
    "\n",
    "# Display all extracted triples using Pandas\n",
    "print(\"\\n--- All Extracted Triples (Before Normalization) ---\")\n",
    "if all_extracted_triples:\n",
    "    all_triples_df = pd.DataFrame(all_extracted_triples)\n",
    "    display(all_triples_df)\n",
    "else:\n",
    "    print(\"No triples were successfully extracted.\")\n",
    "print(\"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "normalization-intro-detail",
   "metadata": {},
   "source": [
    "## Step 8: Normalize and De-duplicate Triples\n",
    "\n",
    "Now, we clean up the extracted triples:\n",
    "1.  **Normalize:** Trim whitespace, convert to lowercase.\n",
    "2.  **Filter:** Remove triples with empty parts after normalization.\n",
    "3.  **De-duplicate:** Remove exact duplicate `(subject, predicate, object)` combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "normalization-init",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting normalization and de-duplication of 18 triples...\n"
     ]
    }
   ],
   "source": [
    "# Initialize lists and tracking variables\n",
    "normalized_triples = []\n",
    "seen_triples = set() # Tracks (subject, predicate, object) tuples\n",
    "original_count = len(all_extracted_triples)\n",
    "empty_removed_count = 0\n",
    "duplicates_removed_count = 0\n",
    "\n",
    "print(f\"Starting normalization and de-duplication of {original_count} triples...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "normalization-loop",
   "metadata": {},
   "source": [
    "We'll iterate through the extracted triples, clean them, and check for duplicates. We'll show the first few transformations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "normalization-process",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing triples for normalization (showing first 5 examples):\n",
      "\n",
      "--- Example 1 ---\n",
      "Original Triple (Chunk 1): {'subject': 'marie curie', 'predicate': 'born as', 'object': 'maria skłodowska', 'chunk': 1}\n",
      "Normalized: SUB='marie curie', PRED='born as', OBJ='maria skłodowska'\n",
      "Status: Kept (New Unique Triple)\n",
      "\n",
      "--- Example 2 ---\n",
      "Original Triple (Chunk 1): {'subject': 'marie curie', 'predicate': 'born in', 'object': 'warsaw, poland', 'chunk': 1}\n",
      "Normalized: SUB='marie curie', PRED='born in', OBJ='warsaw, poland'\n",
      "Status: Kept (New Unique Triple)\n",
      "\n",
      "--- Example 3 ---\n",
      "Original Triple (Chunk 1): {'subject': 'marie curie', 'predicate': 'was', 'object': 'physicist', 'chunk': 1}\n",
      "Normalized: SUB='marie curie', PRED='was', OBJ='physicist'\n",
      "Status: Kept (New Unique Triple)\n",
      "\n",
      "--- Example 4 ---\n",
      "Original Triple (Chunk 1): {'subject': 'marie curie', 'predicate': 'was', 'object': 'chemist', 'chunk': 1}\n",
      "Normalized: SUB='marie curie', PRED='was', OBJ='chemist'\n",
      "Status: Kept (New Unique Triple)\n",
      "\n",
      "--- Example 5 ---\n",
      "Original Triple (Chunk 1): {'subject': 'marie curie', 'predicate': 'conducted', 'object': 'research on radioactivity', 'chunk': 1}\n",
      "Normalized: SUB='marie curie', PRED='conducted', OBJ='research on radioactivity'\n",
      "Status: Kept (New Unique Triple)\n",
      "\n",
      "... Finished processing 18 triples.\n"
     ]
    }
   ],
   "source": [
    "print(\"Processing triples for normalization (showing first 5 examples):\")\n",
    "example_limit = 5\n",
    "processed_count = 0\n",
    "\n",
    "for i, triple in enumerate(all_extracted_triples):\n",
    "    show_example = (i < example_limit)\n",
    "    if show_example:\n",
    "        print(f\"\\n--- Example {i+1} ---\")\n",
    "        print(f\"Original Triple (Chunk {triple.get('chunk', '?')}): {triple}\")\n",
    "        \n",
    "    subject_raw = triple.get('subject')\n",
    "    predicate_raw = triple.get('predicate')\n",
    "    object_raw = triple.get('object')\n",
    "    chunk_num = triple.get('chunk', 'unknown')\n",
    "    \n",
    "    triple_valid = False\n",
    "    normalized_sub, normalized_pred, normalized_obj = None, None, None\n",
    "\n",
    "    if isinstance(subject_raw, str) and isinstance(predicate_raw, str) and isinstance(object_raw, str):\n",
    "        # 1. Normalize\n",
    "        normalized_sub = subject_raw.strip().lower()\n",
    "        normalized_pred = re.sub(r'\\s+', ' ', predicate_raw.strip().lower()).strip()\n",
    "        normalized_obj = object_raw.strip().lower()\n",
    "        if show_example:\n",
    "            print(f\"Normalized: SUB='{normalized_sub}', PRED='{normalized_pred}', OBJ='{normalized_obj}'\")\n",
    "\n",
    "        # 2. Filter Empty\n",
    "        if normalized_sub and normalized_pred and normalized_obj:\n",
    "            triple_identifier = (normalized_sub, normalized_pred, normalized_obj)\n",
    "            \n",
    "            # 3. De-duplicate\n",
    "            if triple_identifier not in seen_triples:\n",
    "                normalized_triples.append({\n",
    "                    'subject': normalized_sub,\n",
    "                    'predicate': normalized_pred,\n",
    "                    'object': normalized_obj,\n",
    "                    'source_chunk': chunk_num\n",
    "                })\n",
    "                seen_triples.add(triple_identifier)\n",
    "                triple_valid = True\n",
    "                if show_example:\n",
    "                    print(\"Status: Kept (New Unique Triple)\")\n",
    "            else:\n",
    "                duplicates_removed_count += 1\n",
    "                if show_example:\n",
    "                    print(\"Status: Discarded (Duplicate)\")\n",
    "        else:\n",
    "            empty_removed_count += 1\n",
    "            if show_example:\n",
    "                print(\"Status: Discarded (Empty component after normalization)\")\n",
    "    else:\n",
    "        empty_removed_count += 1 # Count non-string/missing as needing removal\n",
    "        if show_example:\n",
    "             print(\"Status: Discarded (Non-string or missing component)\")\n",
    "    processed_count += 1\n",
    "\n",
    "print(f\"\\n... Finished processing {processed_count} triples.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "normalization-summary",
   "metadata": {},
   "source": [
    "Let's summarize the normalization results and display the final list of unique, clean triples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "normalization-display",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Normalization & De-duplication Summary ---\n",
      "Original extracted triple count: 18\n",
      "Triples removed (empty/invalid components): 0\n",
      "Duplicate triples removed: 0\n",
      "Final unique, normalized triple count: 18\n",
      "-------------------------\n",
      "\n",
      "--- Final Normalized Triples ---\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subject</th>\n",
       "      <th>predicate</th>\n",
       "      <th>object</th>\n",
       "      <th>source_chunk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born as</td>\n",
       "      <td>maria skłodowska</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born in</td>\n",
       "      <td>warsaw, poland</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>physicist</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>chemist</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>conducted</td>\n",
       "      <td>research on radioactivity</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>discovered</td>\n",
       "      <td>polonium</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>discovered</td>\n",
       "      <td>radium</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>first woman to win nobel prize</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>first person to win nobel prize twice</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>only woman to win nobel prize twice</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>was</td>\n",
       "      <td>only person to win nobel prize in two scientific fields</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>won</td>\n",
       "      <td>nobel prize in physics</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>won</td>\n",
       "      <td>nobel prize in chemistry</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>developed</td>\n",
       "      <td>mobile radiography units</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>died in</td>\n",
       "      <td>1934</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>died from</td>\n",
       "      <td>aplastic anemia</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born on</td>\n",
       "      <td>november 7, 1867</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>born to</td>\n",
       "      <td>family of teachers</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        subject   predicate  \\\n",
       "0   marie curie     born as   \n",
       "1   marie curie     born in   \n",
       "2   marie curie         was   \n",
       "3   marie curie         was   \n",
       "4   marie curie   conducted   \n",
       "5   marie curie  discovered   \n",
       "6   marie curie  discovered   \n",
       "7   marie curie         was   \n",
       "8   marie curie         was   \n",
       "9   marie curie         was   \n",
       "10  marie curie         was   \n",
       "11  marie curie         won   \n",
       "12  marie curie         won   \n",
       "13  marie curie   developed   \n",
       "14  marie curie     died in   \n",
       "15  marie curie   died from   \n",
       "16  marie curie     born on   \n",
       "17  marie curie     born to   \n",
       "\n",
       "                                                     object  source_chunk  \n",
       "0                                          maria skłodowska             1  \n",
       "1                                            warsaw, poland             1  \n",
       "2                                                 physicist             1  \n",
       "3                                                   chemist             1  \n",
       "4                                 research on radioactivity             1  \n",
       "5                                                  polonium             1  \n",
       "6                                                    radium             1  \n",
       "7                            first woman to win nobel prize             1  \n",
       "8                     first person to win nobel prize twice             1  \n",
       "9                       only woman to win nobel prize twice             1  \n",
       "10  only person to win nobel prize in two scientific fields             1  \n",
       "11                                   nobel prize in physics             1  \n",
       "12                                 nobel prize in chemistry             1  \n",
       "13                                 mobile radiography units             1  \n",
       "14                                                     1934             1  \n",
       "15                                          aplastic anemia             1  \n",
       "16                                         november 7, 1867             1  \n",
       "17                                       family of teachers             1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "# --- Summary of Normalization --- \n",
    "print(f\"\\n--- Normalization & De-duplication Summary ---\")\n",
    "print(f\"Original extracted triple count: {original_count}\")\n",
    "print(f\"Triples removed (empty/invalid components): {empty_removed_count}\")\n",
    "print(f\"Duplicate triples removed: {duplicates_removed_count}\")\n",
    "final_count = len(normalized_triples)\n",
    "print(f\"Final unique, normalized triple count: {final_count}\")\n",
    "print(\"-\" * 25)\n",
    "\n",
    "# Display a sample of normalized triples using Pandas\n",
    "print(\"\\n--- Final Normalized Triples ---\")\n",
    "if normalized_triples:\n",
    "    normalized_df = pd.DataFrame(normalized_triples)\n",
    "    display(normalized_df)\n",
    "else:\n",
    "    print(\"No valid triples remain after normalization.\")\n",
    "print(\"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graph-build-intro-detail",
   "metadata": {},
   "source": [
    "## Step 9: Build the Knowledge Graph with NetworkX\n",
    "\n",
    "Using the clean `normalized_triples`, we construct a `networkx` directed graph (`DiGraph`).\n",
    "*   Subjects and Objects become nodes.\n",
    "*   Predicates become edge labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "graph-build-init",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialized an empty NetworkX DiGraph.\n",
      "--- Initial Graph Info ---\n",
      "Type: DiGraph\n",
      "Number of nodes: 0\n",
      "Number of edges: 0\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "# Create an empty directed graph\n",
    "knowledge_graph = nx.DiGraph()\n",
    "\n",
    "print(\"Initialized an empty NetworkX DiGraph.\")\n",
    "# Visualize the initial empty graph state\n",
    "print(\"--- Initial Graph Info ---\")\n",
    "try:\n",
    "    # Try the newer method first\n",
    "    print(nx.info(knowledge_graph))\n",
    "except AttributeError:\n",
    "    # Fallback for different NetworkX versions\n",
    "    print(f\"Type: {type(knowledge_graph).__name__}\")\n",
    "    print(f\"Number of nodes: {knowledge_graph.number_of_nodes()}\")\n",
    "    print(f\"Number of edges: {knowledge_graph.number_of_edges()}\")\n",
    "print(\"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graph-build-loop",
   "metadata": {},
   "source": [
    "Now, we add the triples to the graph one by one, showing the graph's growth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "graph-build-add",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding triples to the NetworkX graph...\n",
      "\n",
      "--- Graph Info after adding Triple #5 --- (marie curie -> research on radioactivity)\n",
      "Type: DiGraph\n",
      "Number of nodes: 6\n",
      "Number of edges: 5\n",
      "\n",
      "--- Graph Info after adding Triple #10 --- (marie curie -> only woman to win nobel prize twice)\n",
      "Type: DiGraph\n",
      "Number of nodes: 11\n",
      "Number of edges: 10\n",
      "\n",
      "--- Graph Info after adding Triple #15 --- (marie curie -> 1934)\n",
      "Type: DiGraph\n",
      "Number of nodes: 16\n",
      "Number of edges: 15\n",
      "\n",
      "--- Graph Info after adding Triple #18 --- (marie curie -> family of teachers)\n",
      "Type: DiGraph\n",
      "Number of nodes: 19\n",
      "Number of edges: 18\n",
      "\n",
      "Finished adding triples. Processed 18 edges.\n"
     ]
    }
   ],
   "source": [
    "print(\"Adding triples to the NetworkX graph...\")\n",
    "\n",
    "added_edges_count = 0\n",
    "update_interval = 5 # How often to print graph info update\n",
    "\n",
    "if not normalized_triples:\n",
    "    print(\"Warning: No normalized triples to add to the graph.\")\n",
    "else:\n",
    "    for i, triple in enumerate(normalized_triples):\n",
    "        subject_node = triple['subject']\n",
    "        object_node = triple['object']\n",
    "        predicate_label = triple['predicate']\n",
    "        \n",
    "        # Nodes are added automatically when adding edges, but explicit calls are fine too\n",
    "        # knowledge_graph.add_node(subject_node) \n",
    "        # knowledge_graph.add_node(object_node)\n",
    "        \n",
    "        # Add the directed edge with the predicate as a 'label' attribute\n",
    "        knowledge_graph.add_edge(subject_node, object_node, label=predicate_label)\n",
    "        added_edges_count += 1\n",
    "        \n",
    "        # --- Visualize Graph Growth --- \n",
    "        if (i + 1) % update_interval == 0 or (i + 1) == len(normalized_triples):\n",
    "            print(f\"\\n--- Graph Info after adding Triple #{i+1} --- ({subject_node} -> {object_node})\")\n",
    "            try:\n",
    "                # Try the newer method first\n",
    "                print(nx.info(knowledge_graph))\n",
    "            except AttributeError:\n",
    "                # Fallback for different NetworkX versions\n",
    "                print(f\"Type: {type(knowledge_graph).__name__}\")\n",
    "                print(f\"Number of nodes: {knowledge_graph.number_of_nodes()}\")\n",
    "                print(f\"Number of edges: {knowledge_graph.number_of_edges()}\")\n",
    "            # For very large graphs, printing info too often can be slow. Adjust interval.\n",
    "\n",
    "print(f\"\\nFinished adding triples. Processed {added_edges_count} edges.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "graph-build-summary",
   "metadata": {},
   "source": [
    "Let's look at the final graph statistics and sample nodes/edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "graph-build-stats",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Final NetworkX Graph Summary ---\n",
      "Total unique nodes (entities): 19\n",
      "Total unique edges (relationships): 18\n",
      "Graph density: 0.0526\n",
      "The graph is weakly connected (all nodes reachable ignoring direction).\n",
      "-------------------------\n",
      "\n",
      "--- Sample Nodes (First 10) ---\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Node Sample</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>marie curie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>maria skłodowska</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>warsaw, poland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>physicist</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>chemist</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>research on radioactivity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>polonium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>radium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>first woman to win nobel prize</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>first person to win nobel prize twice</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Node Sample\n",
       "0                            marie curie\n",
       "1                       maria skłodowska\n",
       "2                         warsaw, poland\n",
       "3                              physicist\n",
       "4                                chemist\n",
       "5              research on radioactivity\n",
       "6                               polonium\n",
       "7                                 radium\n",
       "8         first woman to win nobel prize\n",
       "9  first person to win nobel prize twice"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--- Sample Edges (First 10 with Labels) ---\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Source</th>\n",
       "      <th>Target</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>maria skłodowska</td>\n",
       "      <td>born as</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>warsaw, poland</td>\n",
       "      <td>born in</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>physicist</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>chemist</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>research on radioactivity</td>\n",
       "      <td>conducted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>polonium</td>\n",
       "      <td>discovered</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>radium</td>\n",
       "      <td>discovered</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>first woman to win nobel prize</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>first person to win nobel prize twice</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>marie curie</td>\n",
       "      <td>only woman to win nobel prize twice</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Source                                 Target       Label\n",
       "0  marie curie                       maria skłodowska     born as\n",
       "1  marie curie                         warsaw, poland     born in\n",
       "2  marie curie                              physicist         was\n",
       "3  marie curie                                chemist         was\n",
       "4  marie curie              research on radioactivity   conducted\n",
       "5  marie curie                               polonium  discovered\n",
       "6  marie curie                                 radium  discovered\n",
       "7  marie curie         first woman to win nobel prize         was\n",
       "8  marie curie  first person to win nobel prize twice         was\n",
       "9  marie curie    only woman to win nobel prize twice         was"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "# --- Final Graph Statistics --- \n",
    "num_nodes = knowledge_graph.number_of_nodes()\n",
    "num_edges = knowledge_graph.number_of_edges()\n",
    "\n",
    "print(f\"\\n--- Final NetworkX Graph Summary ---\")\n",
    "print(f\"Total unique nodes (entities): {num_nodes}\")\n",
    "print(f\"Total unique edges (relationships): {num_edges}\")\n",
    "\n",
    "if num_edges != added_edges_count and isinstance(knowledge_graph, nx.DiGraph):\n",
    "     print(f\"Note: Added {added_edges_count} edges, but graph has {num_edges}. DiGraph overwrites edges with same source/target. Use MultiDiGraph if multiple edges needed.\")\n",
    "\n",
    "if num_nodes > 0:\n",
    "    try:\n",
    "       density = nx.density(knowledge_graph)\n",
    "       print(f\"Graph density: {density:.4f}\")\n",
    "       if nx.is_weakly_connected(knowledge_graph):\n",
    "           print(\"The graph is weakly connected (all nodes reachable ignoring direction).\")\n",
    "       else:\n",
    "           num_components = nx.number_weakly_connected_components(knowledge_graph)\n",
    "           print(f\"The graph has {num_components} weakly connected components.\")\n",
    "    except Exception as e:\n",
    "        print(f\"Could not calculate some graph metrics: {e}\") # Handle potential errors on empty/small graphs\n",
    "else:\n",
    "    print(\"Graph is empty, cannot calculate metrics.\")\n",
    "print(\"-\" * 25)\n",
    "\n",
    "# --- Sample Nodes --- \n",
    "print(\"\\n--- Sample Nodes (First 10) ---\")\n",
    "if num_nodes > 0:\n",
    "    nodes_sample = list(knowledge_graph.nodes())[:10]\n",
    "    display(pd.DataFrame(nodes_sample, columns=['Node Sample']))\n",
    "else:\n",
    "    print(\"Graph has no nodes.\")\n",
    "\n",
    "# --- Sample Edges --- \n",
    "print(\"\\n--- Sample Edges (First 10 with Labels) ---\")\n",
    "if num_edges > 0:\n",
    "    edges_sample = []\n",
    "    for u, v, data in list(knowledge_graph.edges(data=True))[:10]:\n",
    "        edges_sample.append({'Source': u, 'Target': v, 'Label': data.get('label', 'N/A')})\n",
    "    display(pd.DataFrame(edges_sample))\n",
    "else:\n",
    "    print(\"Graph has no edges.\")\n",
    "print(\"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viz-intro-ipy-detail",
   "metadata": {},
   "source": [
    "## Step 10: Visualize the Graph Interactively with ipycytoscape\n",
    "\n",
    "Finally, we visualize the constructed graph interactively within the notebook using `ipycytoscape`. We'll convert the `networkx` data, define styles, and display the widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "viz-check-graph",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing interactive visualization...\n",
      "Graph seems valid for visualization (19 nodes, 18 edges).\n"
     ]
    }
   ],
   "source": [
    "print(\"Preparing interactive visualization...\")\n",
    "\n",
    "# --- Check Graph Validity for Visualization --- \n",
    "can_visualize = False\n",
    "if 'knowledge_graph' not in locals() or not isinstance(knowledge_graph, nx.Graph):\n",
    "    print(\"Error: 'knowledge_graph' not found or is not a NetworkX graph.\")\n",
    "elif knowledge_graph.number_of_nodes() == 0:\n",
    "    print(\"NetworkX Graph is empty. Cannot visualize.\")\n",
    "else:\n",
    "    print(f\"Graph seems valid for visualization ({knowledge_graph.number_of_nodes()} nodes, {knowledge_graph.number_of_edges()} edges).\")\n",
    "    can_visualize = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viz-data-convert",
   "metadata": {},
   "source": [
    "### 10.1 Convert NetworkX Data to Cytoscape Format\n",
    "\n",
    "`ipycytoscape` requires nodes and edges in a specific JSON-like format (list of dictionaries)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "viz-convert-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converting nodes...\n",
      "Converted 19 nodes.\n",
      "Converting edges...\n",
      "Converted 18 edges.\n",
      "\n",
      "--- Sample Cytoscape Node Data (First 2) ---\n",
      "[\n",
      "  {\n",
      "    \"data\": {\n",
      "      \"id\": \"marie curie\",\n",
      "      \"label\": \"marie\\ncurie\",\n",
      "      \"degree\": 18,\n",
      "      \"size\": 65.0,\n",
      "      \"tooltip_text\": \"Entity: marie curie\\nDegree: 18\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"data\": {\n",
      "      \"id\": \"maria sk\\u0142odowska\",\n",
      "      \"label\": \"maria\\nsk\\u0142odowska\",\n",
      "      \"degree\": 1,\n",
      "      \"size\": 17.77777777777778,\n",
      "      \"tooltip_text\": \"Entity: maria sk\\u0142odowska\\nDegree: 1\"\n",
      "    }\n",
      "  }\n",
      "]\n",
      "\n",
      "--- Sample Cytoscape Edge Data (First 2) ---\n",
      "[\n",
      "  {\n",
      "    \"data\": {\n",
      "      \"id\": \"edge_0\",\n",
      "      \"source\": \"marie curie\",\n",
      "      \"target\": \"maria sk\\u0142odowska\",\n",
      "      \"label\": \"born as\",\n",
      "      \"tooltip_text\": \"Relationship: born as\"\n",
      "    }\n",
      "  },\n",
      "  {\n",
      "    \"data\": {\n",
      "      \"id\": \"edge_1\",\n",
      "      \"source\": \"marie curie\",\n",
      "      \"target\": \"warsaw, poland\",\n",
      "      \"label\": \"born in\",\n",
      "      \"tooltip_text\": \"Relationship: born in\"\n",
      "    }\n",
      "  }\n",
      "]\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "cytoscape_nodes = []\n",
    "cytoscape_edges = []\n",
    "\n",
    "if can_visualize:\n",
    "    print(\"Converting nodes...\")\n",
    "    # Calculate degrees for node sizing\n",
    "    node_degrees = dict(knowledge_graph.degree())\n",
    "    max_degree = max(node_degrees.values()) if node_degrees else 1\n",
    "    \n",
    "    for node_id in knowledge_graph.nodes():\n",
    "        degree = node_degrees.get(node_id, 0)\n",
    "        # Simple scaling for node size (adjust logic as needed)\n",
    "        node_size = 15 + (degree / max_degree) * 50 if max_degree > 0 else 15\n",
    "        \n",
    "        cytoscape_nodes.append({\n",
    "            'data': {\n",
    "                'id': str(node_id), # ID must be string\n",
    "                'label': str(node_id).replace(' ', '\\n'), # Display label (wrap spaces)\n",
    "                'degree': degree,\n",
    "                'size': node_size,\n",
    "                'tooltip_text': f\"Entity: {str(node_id)}\\nDegree: {degree}\" # Tooltip on hover\n",
    "            }\n",
    "        })\n",
    "    print(f\"Converted {len(cytoscape_nodes)} nodes.\")\n",
    "    \n",
    "    print(\"Converting edges...\")\n",
    "    edge_count = 0\n",
    "    for u, v, data in knowledge_graph.edges(data=True):\n",
    "        edge_id = f\"edge_{edge_count}\" # Unique edge ID\n",
    "        predicate_label = data.get('label', '')\n",
    "        cytoscape_edges.append({\n",
    "            'data': {\n",
    "                'id': edge_id,\n",
    "                'source': str(u),\n",
    "                'target': str(v),\n",
    "                'label': predicate_label, # Label on edge\n",
    "                'tooltip_text': f\"Relationship: {predicate_label}\" # Tooltip on hover\n",
    "            }\n",
    "        })\n",
    "        edge_count += 1\n",
    "    print(f\"Converted {len(cytoscape_edges)} edges.\")\n",
    "    \n",
    "    # Combine into the final structure\n",
    "    cytoscape_graph_data = {'nodes': cytoscape_nodes, 'edges': cytoscape_edges}\n",
    "    \n",
    "    # Visualize the converted structure (first few nodes/edges)\n",
    "    print(\"\\n--- Sample Cytoscape Node Data (First 2) ---\")\n",
    "    print(json.dumps(cytoscape_graph_data['nodes'][:2], indent=2))\n",
    "    print(\"\\n--- Sample Cytoscape Edge Data (First 2) ---\")\n",
    "    print(json.dumps(cytoscape_graph_data['edges'][:2], indent=2))\n",
    "    print(\"-\" * 25)\n",
    "else:\n",
    "     print(\"Skipping data conversion as graph is not valid for visualization.\")\n",
    "     cytoscape_graph_data = {'nodes': [], 'edges': []}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viz-widget-create",
   "metadata": {},
   "source": [
    "### 10.2 Create and Configure the Cytoscape Widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "viz-create-widget-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating ipycytoscape widget...\n",
      "Widget created.\n",
      "Loading graph data into widget...\n",
      "Data loaded.\n"
     ]
    }
   ],
   "source": [
    "if can_visualize:\n",
    "    print(\"Creating ipycytoscape widget...\")\n",
    "    cyto_widget = ipycytoscape.CytoscapeWidget()\n",
    "    print(\"Widget created.\")\n",
    "    \n",
    "    print(\"Loading graph data into widget...\")\n",
    "    cyto_widget.graph.add_graph_from_json(cytoscape_graph_data, directed=True)\n",
    "    print(\"Data loaded.\")\n",
    "else:\n",
    "    print(\"Skipping widget creation.\")\n",
    "    cyto_widget = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viz-styling",
   "metadata": {},
   "source": [
    "### 10.3 Define Visual Style\n",
    "\n",
    "We use a CSS-like syntax to control the appearance of nodes and edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "viz-style-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defining enhanced colorful and interactive visual style...\n",
      "Setting enhanced visual style on widget...\n",
      "Enhanced colorful and interactive style applied successfully.\n"
     ]
    }
   ],
   "source": [
    "if cyto_widget:\n",
    "    print(\"Defining enhanced colorful and interactive visual style...\")\n",
    "    # More vibrant and colorful styling with a modern color scheme\n",
    "    visual_style = [\n",
    "        {\n",
    "            'selector': 'node',\n",
    "            'style': {\n",
    "                'label': 'data(label)',\n",
    "                'width': 'data(size)',\n",
    "                'height': 'data(size)',\n",
    "                'background-color': '#3498db',  # Bright blue\n",
    "                'background-opacity': 0.9,\n",
    "                'color': '#ffffff',             # White text\n",
    "                'font-size': '12px',\n",
    "                'font-weight': 'bold',\n",
    "                'text-valign': 'center',\n",
    "                'text-halign': 'center',\n",
    "                'text-wrap': 'wrap',\n",
    "                'text-max-width': '100px',\n",
    "                'text-outline-width': 2,\n",
    "                'text-outline-color': '#2980b9',  # Matching outline\n",
    "                'text-outline-opacity': 0.7,\n",
    "                'border-width': 3,\n",
    "                'border-color': '#1abc9c',      # Turquoise border\n",
    "                'border-opacity': 0.9,\n",
    "                'shape': 'ellipse',\n",
    "                'transition-property': 'background-color, border-color, border-width, width, height',\n",
    "                'transition-duration': '0.3s',\n",
    "                'tooltip-text': 'data(tooltip_text)'\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'selector': 'node:selected',\n",
    "            'style': {\n",
    "                'background-color': '#e74c3c',  # Pomegranate red\n",
    "                'border-width': 4,\n",
    "                'border-color': '#c0392b',\n",
    "                'text-outline-color': '#e74c3c',\n",
    "                'width': 'data(size) * 1.2',    # Enlarge selected nodes\n",
    "                'height': 'data(size) * 1.2'\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'selector': 'node:hover',\n",
    "            'style': {\n",
    "                'background-color': '#9b59b6',  # Purple on hover\n",
    "                'border-width': 4,\n",
    "                'border-color': '#8e44ad',\n",
    "                'cursor': 'pointer',\n",
    "                'z-index': 999\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'selector': 'edge',\n",
    "            'style': {\n",
    "                'label': 'data(label)',\n",
    "                'width': 2.5,\n",
    "                'curve-style': 'bezier',\n",
    "                'line-color': '#2ecc71',         # Green\n",
    "                'line-opacity': 0.8,\n",
    "                'target-arrow-color': '#27ae60',\n",
    "                'target-arrow-shape': 'triangle',\n",
    "                'arrow-scale': 1.5,\n",
    "                'font-size': '10px',\n",
    "                'font-weight': 'normal',\n",
    "                'color': '#2c3e50',\n",
    "                'text-background-opacity': 0.9,\n",
    "                'text-background-color': '#ecf0f1',\n",
    "                'text-background-shape': 'roundrectangle',\n",
    "                'text-background-padding': '3px',\n",
    "                'text-rotation': 'autorotate',\n",
    "                'edge-text-rotation': 'autorotate',\n",
    "                'transition-property': 'line-color, width, target-arrow-color',\n",
    "                'transition-duration': '0.3s',\n",
    "                'tooltip-text': 'data(tooltip_text)'\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'selector': 'edge:selected',\n",
    "            'style': {\n",
    "                'line-color': '#f39c12',         # Yellow-orange\n",
    "                'target-arrow-color': '#d35400',\n",
    "                'width': 4,\n",
    "                'text-background-color': '#f1c40f',\n",
    "                'color': '#ffffff',               # White text\n",
    "                'z-index': 998\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'selector': 'edge:hover',\n",
    "            'style': {\n",
    "                'line-color': '#e67e22',         # Orange on hover\n",
    "                'width': 3.5,\n",
    "                'cursor': 'pointer',\n",
    "                'target-arrow-color': '#d35400',\n",
    "                'z-index': 997\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'selector': '.center-node',\n",
    "            'style': {\n",
    "                'background-color': '#16a085',    # Teal\n",
    "                'background-opacity': 1,\n",
    "                'border-width': 4,\n",
    "                'border-color': '#1abc9c',        # Turquoise border\n",
    "                'border-opacity': 1\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "    \n",
    "    print(\"Setting enhanced visual style on widget...\")\n",
    "    cyto_widget.set_style(visual_style)\n",
    "    \n",
    "    # Apply a better animated layout\n",
    "    cyto_widget.set_layout(name='cose', \n",
    "                          nodeRepulsion=5000, \n",
    "                          nodeOverlap=40, \n",
    "                          idealEdgeLength=120, \n",
    "                          edgeElasticity=200, \n",
    "                          nestingFactor=6, \n",
    "                          gravity=90, \n",
    "                          numIter=2500,\n",
    "                          animate=True,\n",
    "                          animationDuration=1000,\n",
    "                          initialTemp=300,\n",
    "                          coolingFactor=0.95)\n",
    "    \n",
    "    # Add a special class to main nodes (Marie Curie)\n",
    "    if len(cyto_widget.graph.nodes) > 0:\n",
    "        main_nodes = [node.data['id'] for node in cyto_widget.graph.nodes \n",
    "                     if node.data.get('degree', 0) > 10]\n",
    "        \n",
    "        # Create gradient styles for center nodes\n",
    "        for i, node_id in enumerate(main_nodes):\n",
    "            # Use vibrant colors for center nodes\n",
    "            center_style = {\n",
    "                'selector': f'node[id = \"{node_id}\"]',\n",
    "                'style': {\n",
    "                    'background-color': '#9b59b6',   # Purple\n",
    "                    'background-opacity': 0.95,\n",
    "                    'border-width': 4,\n",
    "                    'border-color': '#8e44ad',      # Darker purple border\n",
    "                    'border-opacity': 1,\n",
    "                    'text-outline-width': 3,\n",
    "                    'text-outline-color': '#8e44ad',\n",
    "                    'font-size': '14px'\n",
    "                }\n",
    "            }\n",
    "            visual_style.append(center_style)\n",
    "        \n",
    "        # Update the style with the new additions\n",
    "        cyto_widget.set_style(visual_style)\n",
    "    \n",
    "    print(\"Enhanced colorful and interactive style applied successfully.\")\n",
    "else:\n",
    "    print(\"Skipping style definition.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viz-layout",
   "metadata": {},
   "source": [
    "### 10.4 Set Layout Algorithm\n",
    "\n",
    "We choose an algorithm to automatically arrange the nodes and edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "viz-layout-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting layout algorithm ('cose')...\n",
      "Layout set. The graph will arrange itself when displayed.\n"
     ]
    }
   ],
   "source": [
    "if cyto_widget:\n",
    "    print(\"Setting layout algorithm ('cose')...\")\n",
    "    # cose (Compound Spring Embedder) is often good for exploring connections\n",
    "    cyto_widget.set_layout(name='cose', \n",
    "                           animate=True, \n",
    "                           # Adjust parameters for better spacing/layout\n",
    "                           nodeRepulsion=4000, # Increase repulsion \n",
    "                           nodeOverlap=40,    # Increase overlap avoidance\n",
    "                           idealEdgeLength=120, # Slightly longer ideal edges\n",
    "                           edgeElasticity=150, \n",
    "                           nestingFactor=5, \n",
    "                           gravity=100,        # Increase gravity slightly\n",
    "                           numIter=1500,      # More iterations\n",
    "                           initialTemp=200,\n",
    "                           coolingFactor=0.95,\n",
    "                           minTemp=1.0)\n",
    "    print(\"Layout set. The graph will arrange itself when displayed.\")\n",
    "else:\n",
    "     print(\"Skipping layout setting.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viz-display",
   "metadata": {},
   "source": [
    "### 10.5 Display the Interactive Graph\n",
    "\n",
    "The final step is to render the interactive widget in the notebook output below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "viz-display-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying interactive graph widget below...\n",
      "Interact: Zoom (scroll), Pan (drag background), Move Nodes (drag nodes), Hover for details.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57a8d1dc4a6448b88f57950762f23b05",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "CytoscapeWidget(cytoscape_layout={'name': 'cose', 'nodeRepulsion': 4000, 'nodeOverlap': 40, 'idealEdgeLength':…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-------------------------\n",
      "End of Visualization Step.\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "if cyto_widget:\n",
    "    print(\"Displaying interactive graph widget below...\")\n",
    "    print(\"Interact: Zoom (scroll), Pan (drag background), Move Nodes (drag nodes), Hover for details.\")\n",
    "    display(cyto_widget)\n",
    "else:\n",
    "    print(\"No widget to display.\")\n",
    "\n",
    "# Add a clear separator\n",
    "print(\"\\n\" + \"-\" * 25 + \"\\nEnd of Visualization Step.\" + \"\\n\" + \"-\" * 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conclusion",
   "metadata": {},
   "source": [
    "## Step 11: Conclusion and Next Steps\n",
    "\n",
    "We have now walked through a very granular process:\n",
    "1.  Setup libraries and LLM connection.\n",
    "2.  Defined and chunked input text, visualizing intermediate steps.\n",
    "3.  Defined detailed prompts for the LLM.\n",
    "4.  Iterated through chunks (demonstrated with one), showing raw LLM output, parsed JSON, and extracted triples for each.\n",
    "5.  Aggregated, normalized, and de-duplicated triples, showing the results.\n",
    "6.  Built the `networkx` graph step-by-step, showing its growth.\n",
    "7.  Converted data for `ipycytoscape` and visualized the final interactive knowledge graph directly in the notebook.\n",
    "\n",
    "This detailed breakdown should make the transformation from unstructured text to a structured, visual knowledge graph much clearer.\n",
    "\n",
    "**Potential Improvements and Further Exploration:**\n",
    "*   **Run Full Loop:** Execute the LLM extraction and normalization across *all* chunks for a complete graph.\n",
    "*   **Advanced Normalization:** Implement entity linking or relationship clustering.\n",
    "*   **Error Handling:** Add retries for LLM calls, better handling of persistent chunk failures.\n",
    "*   **Prompt Tuning:** Experiment with different models, prompts, and parameters.\n",
    "*   **Evaluation:** Assess the quality of extracted triples (Precision/Recall).\n",
    "*   **Richer Visualization:** Use node types for colors/shapes, add community detection coloring, implement more interactive features using ipycytoscape callbacks.\n",
    "*   **Graph Analysis:** Apply `networkx` algorithms (centrality, paths, etc.).\n",
    "*   **Persistence:** Store results in a graph database (Neo4j, etc.)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv-kg",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
